{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Getting started with TensorFlow (Graph Mode)\n",
    "\n",
    "**Learning Objectives**\n",
    "  - Understand the difference between Tensorflow's two modes: Eager Execution and Graph Execution\n",
    "  - Get used to deferred execution paradigm: first define a graph then run it in a `tf.Session()`\n",
    "  - Understand how to parameterize a graph using `tf.placeholder()` and `feed_dict`\n",
    "  - Understand the difference between constant Tensors and variable Tensors, and how to define each\n",
    "  - Practice using mid-level `tf.train` module for gradient descent\n",
    "\n",
    "## Introduction\n",
    "\n",
    "**Eager Execution**\n",
    "\n",
    "Eager mode evaluates operations immediatley and return concrete values immediately. To enable eager mode simply place `tf.enable_eager_execution()` at the top of your code. We recommend using eager execution when prototyping as it is intuitive, easier to debug, and requires less boilerplate code.\n",
    "\n",
    "**Graph Execution**\n",
    "\n",
    "Graph mode is TensorFlow's default execution mode (although it will change to eager in TF 2.0). In graph mode operations only produce a symbolic graph which doesn't get executed until run within the context of a tf.Session(). This style of coding is less inutitive and has more boilerplate, however it can lead to performance optimizations and is particularly suited for distributing training across multiple devices. We recommend using delayed execution for performance sensitive production code. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "print(tf.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Graph Execution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Adding Two Tensors "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Build the Graph\n",
    "\n",
    "Unlike eager mode, no concrete value will be returned yet. Just a name, shape and type are printed. Behind the scenes a directed graph is being created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = tf.constant(value = [5, 3, 8], dtype = tf.int32)\n",
    "b = tf.constant(value = [3, -1, 2], dtype = tf.int32)\n",
    "c = tf.add(x = a, y = b)\n",
    "print(c)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Run the Graph\n",
    "\n",
    "A graph can be executed in the context of a `tf.Session()`. Think of a session as the bridge between the front-end Python API and the back-end C++ execution engine. \n",
    "\n",
    "Within a session, passing a tensor operation to `run()` will cause Tensorflow to execute all upstream operations in the graph required to calculate that value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with tf.Session() as sess:\n",
    "    result = sess.run(fetches = c)\n",
    "    print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Can you mix eager and graph execution together?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Parameterizing the Grpah \n",
    "\n",
    "What if values of `a` and `b` keep changing? How can you parameterize them so they can be fed in at runtime? \n",
    "\n",
    "*Step 1: Define Placeholders*\n",
    "\n",
    "Define `a` and `b` using `tf.placeholder()`. You'll need to specify the data type of the placeholder, and optionally a tensor shape.\n",
    "\n",
    "*Step 2: Provide feed_dict*\n",
    "\n",
    "Now when invoking `run()` within the `tf.Session()`, in addition to providing a tensor operation to evaluate, you also provide a dictionary whose keys are the names of the placeholders. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = tf.placeholder(dtype = tf.int32, shape = [None])  \n",
    "b = tf.placeholder(dtype = tf.int32, shape = [None])\n",
    "c = tf.add(x = a, y = b)\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    result = sess.run(fetches = c, feed_dict = {\n",
    "        a: [3, 4, 5],\n",
    "        b: [-1, 2, 3]\n",
    "    })\n",
    "    print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Linear Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Toy Dataset\n",
    "We'll model the following:\n",
    "\n",
    "\\begin{equation}\n",
    "y= 2x + 10\n",
    "\\end{equation}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = tf.constant(value = [1,2,3,4,5,6,7,8,9,10], dtype = tf.float32)\n",
    "Y = 2 * X + 10\n",
    "print(\"X:{}\".format(X))\n",
    "print(\"Y:{}\".format(Y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Loss Function\n",
    "Using mean squared error, our loss function is:\n",
    "\\begin{equation}\n",
    "MSE = \\frac{1}{m}\\sum_{i=1}^{m}(\\hat{Y}_i-Y_i)^2\n",
    "\\end{equation}\n",
    "\n",
    "$\\hat{Y}$ represents the vector containing our model's predictions:\n",
    "\\begin{equation}\n",
    "\\hat{Y} = w_0X + w_1\n",
    "\\end{equation}\n",
    "\n",
    "Note below we introduce TF variables for the first time. Unlike constants, variables are mutable. \n",
    "\n",
    "Browse the official TensorFlow [guide on variables](https://www.tensorflow.org/guide/variables) for more information on when/how to use them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### **Exercise 1**\n",
    "\n",
    "Becauase the parameters $w_0$ and $w_1$ will be updated through gradient descent, we need to tell Tensorflow that these values are variables and initialize them accordingly. Have a look at the [Tensorflow usage for variables here](https://www.tensorflow.org/api_docs/python/tf/get_variable). Complete the code below to define and initialize the variables `w0` and `w1`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "with tf.variable_scope(name_or_scope = \"training\", reuse = tf.AUTO_REUSE):\n",
    "    w0 = # TODO: Your code goes here\n",
    "    w1 = # TODO: Your code goes here\n",
    "    \n",
    "Y_hat = w0 * X + w1\n",
    "loss_mse = tf.reduce_mean(input_tensor = (Y_hat - Y)**2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Optimizer\n",
    "\n",
    "An optimizer in TensorFlow both calculates gradients and updates weights. In addition to basic  gradient descent, TF provides implementations of several more advanced optimizers such as ADAM and FTRL. They can all be found in the [tf.train](https://www.tensorflow.org/versions/r1.15/api_docs/python/tf/train) module. \n",
    "\n",
    "Note below we're not expclictly telling the optimizer which tensors are our weight tensors. So how does it know what to update? Optimizers will update all variables in the `tf.GraphKeys.TRAINABLE_VARIABLES` [collection](https://www.tensorflow.org/guide/variables#variable_collections). All variables are added to this collection by default. Since our only variables are `w0` and `w1`, this is the behavior we want. If we had a variable that we *didn't* want to be added to the collection we would set `trainable=false` when creating it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### **Exercise 2**\n",
    "\n",
    "When performing gradient descent, we must specify the learning rate and which optimizer to use. In the training loop we will create below, we'll pass the learning rate to the optimzer using a feed dictionary. Thus, we need to create a placeholder for the value of the learning rate. You can read more about [placeholders in Tensorflow here](https://www.tensorflow.org/api_docs/python/tf/placeholder). Placeholders are used for values that will be fed to the operation later. Complete the code below to create a placeholder for the learning rate.\n",
    "\n",
    "We also want to specify the optimizer for the training loop we'll perform below. There are Tensorflow implementations of various optimizers. Complete the code below to create an optimizer. You can find the available optimizers in the [tf.train Module](https://www.tensorflow.org/versions/r1.15/api_docs/python/tf/train)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "LEARNING_RATE = # TODO: Your code goes here\n",
    "optimizer = # TODO: Your code goes here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Training Loop\n",
    "\n",
    "Note our results are identical to what we found in Eager mode."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### **Exercise 3**\n",
    "\n",
    "Finally we are ready to evaluate our training loop in Graph mode. As before, we need to calculate the gradients and update the weights via our optimizer. Complete the code below to call the optimizer using `sess.run`. You can read more about [using `tf.Session()` to execute operations here](https://www.tensorflow.org/guide/graphs#using_tfsessionrun_to_execute_operations). Note that you will need to also pass a `feed_dict` to specify the learning rate of the optimizer you created above. \n",
    "\n",
    "After completing this Exercise, compare with the training loop we made for Eager mode in the previous lab."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "STEPS = 1000\n",
    "with tf.Session() as sess:\n",
    "    sess.run(tf.global_variables_initializer()) # initialize variables\n",
    "    \n",
    "    for step in range(STEPS):\n",
    "        #1. Calculate gradients and update weights \n",
    "        # TODO: Your code goes here\n",
    "        \n",
    "        #2. Periodically print MSE\n",
    "        if step % 100 == 0:\n",
    "            print(\"STEP: {} MSE: {}\".format(step, sess.run(fetches = loss_mse)))\n",
    "    \n",
    "    # Print final MSE and weights\n",
    "    print(\"STEP: {} MSE: {}\".format(STEPS, sess.run(loss_mse)))\n",
    "    print(\"w0:{}\".format(round(float(sess.run(w0)), 4)))\n",
    "    print(\"w1:{}\".format(round(float(sess.run(w1)), 4)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright 2019 Google Inc. Licensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
